System and method of integrating live video into a contextual background

ABSTRACT

A live video feed is seamlessly inserted into a static background image while maintaining the proper positioning and scaling of the combined image. The current frame of the live video feed is first texturized onto a surface. The surface is a set of points that are translated to the location, e.g., latitude/longitude coordinates, of the live video source. A series of matrix transformations of the texturized surface coordinates is performed. First, the texturized surface is rotated to be tangential to the earth at the location of the live video source. Next, the texturized surface is rotated again to reflect the heading of the live video source. Lastly, the texturized surface is scaled to conform to the background image. The transformed live image is then inserted into the larger background image in its proper position and displayed.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present invention is related to, and claims the benefit of, U.S.Provisional Patent Application No. 60/244,053 filed Oct. 27, 2000entitled, “Real-Time Video Integration with Mapping Software”.

This invention was made with U.S. Government support under Navy contractNo. N0024-98-D-8124. The U.S. Government has certain rights in thisinvention.

FIELD OF THE INVENTION

The present invention relates generally to a system and method forintegrating live video into a contextual background. More specifically,the present invention relates generally to a system and method forintegrating live aerial video onto a broader background map in order toprovide a larger context for the live video.

BACKGROUND OF THE INVENTION

There are many uses for live video that include military applications,search and rescue, and general surveillance among others. Whilereceiving a live video feed certainly provides much useful informationfor the task at hand, often it is difficult to interpret the datawithout having a broader perspective. For instance, consider a searchand rescue mission that is utilizing aerial video to assist in finding alost person. The field of view of the camera is limited. The ability todynamically insert the live video into a broader background imagegreatly enhances the useful information. With a larger background,anything of interest that is spotted by the live video can instantly becorrelated to any number of landmarks that are outside the view of thecamera. This can allow for more efficient rescue operations if, forinstance, the location can be immediately put into a larger perspectivethat includes the roads that will lead to the person the quickest.

Current existing products for displaying a video feed over another imageare not nearly robust enough to allow for a dynamic repositioning of thecamera providing the video feed.

What has not yet been achieved is a system or method that displays abackground image that contains a smaller live image placed over aportion of the larger background image such that the live image iscorrectly positioned and scaled with respect to the background image.

Thus, what is needed is a system and method that can display an image ofa live picture that is placed correctly relative to a background image.Moreover, the system should be dynamic in nature allowing the livepicture to be moved around the background image in real-time.

SUMMARY OF THE INVENTION

In order to seamlessly insert a live video feed into a static backgroundimage while maintaining the proper positioning and scaling of thecombined image the live video feed is first texturized onto a surface.The surface is a set of points that are translated to the location,e.g., latitude/longitude coordinates, of the live video source. A seriesof matrix transformations of the texturized surface coordinates isperformed. First, the texturized surface is rotated to be tangential tothe earth at that location. The location information is synchronizedwith the live video feed and is passed along with the live video feed ina position information signal to the processing device performing thecoordinate transformations. Next, the texturized surface is rotatedagain to reflect the heading of the live video source. The headinginformation, including pitch, yaw, and roll, etc. of the camera is alsoincluded in the position information signal that accompanies the livevideo feed. Lastly, the texturized surface is scaled to conform to thebackground image and the transformed live image is inserted into thelarger background image in its proper position.

Other aspects and features of the present invention will become apparentto those ordinarily skilled in the art upon review of the followingdescription of specific embodiments of the invention in conjunction withthe accompanying figures.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a flowchart describing the steps comprising the presentinvention.

FIG. 2 is a sample screen display showing a live video feed integratedinto a contextual map background.

DETAILED DISCLOSURE OF THE INVENTION

The insertion of a smaller image into a larger image gives a newperspective to the smaller image in that it can be viewed in the contextof a much larger picture. The new perspective provides additionalinformation not available by viewing just the smaller image alone. Inorder to seamlessly insert a live video feed into a larger backgroundimage, the live video feed must be processed so that it can be properlypositioned within the background image. The goal is to allow the smallerimage to be properly positioned within the larger image and to allow thesmaller image to be moved seamlessly and in real-time all over thebackground image.

The description herein relates to a typical application of the presentinvention that involves an unmanned aerial vehicle (UAV) having a videosource (camera) mounted thereon. The UAV flies over an area providinglive video via its camera to a remotely situated display viewable by asystem operator. This application is merely illustrative as it providesan environment within which to describe the processes performed by thepresent invention. Those of ordinary skill in the art can readily deviseother applications without departing from the spirit or scope of thepresent invention.

FIG. 1 is a flowchart describing the steps comprising the presentinvention. The present invention utilizes a double buffer system. Eachbuffer includes data for both the current live video frame and thebackground image. One buffer is subjected to a series of processingsteps and upon completion of the processing, it is swapped to thedisplay screen. Double buffering allows for a more eye appealing screendisplay that is not disjointed. Each frame of the live video feed isindividually processed and the screen display is refreshed for eachframe.

The first step is to texture map or “texturize” the current live videoframe 10 onto a surface. Texture mapping is a well-known process thatinvolves taking a picture and drawing it onto a polygon such as atriangle. There are many texture mapping processes that currently exist.One embodiment of the present invention uses the Silicon Graphics“OpenGL” set of tools to perform texture mapping and coordinatetransformations. The next step is to translate the texturized frame sothat it is centered on the viewpoint of the video source (camera) 12.The video frame is then bound to the texturized image 14.

A series of matrix transformations are performed on the texturized image16 in order to make the live video frame seamlessly insertable into thelarger background image. The first matrix transformation involvestranslating the texturized frame to the correct position relative to thebackground image 16 a. In order to perform this step, the live videofeed from the video source must be synchronized with precise informationpertaining to the current status of the video source. Status informationincludes, but is not limited to, orientation, pitch, roll, and zoom.Mere position and heading information is insufficient, because imagejitter introduced as the video source turns or is buffeted by windcauses the video source to no longer point straight down. Withoutprecise position information, the live video and the background imagemap will not be synchronized.

The next transformation is to rotate the texturized frame so it istangent to the background image 16 b. In other words, the texturizedimage is warped to conform to the surface of the background image. Thisis followed by rotating the texturized image again so that it is alignedwith the current heading of the video source 16 c. Lastly, thetexturized image is scaled to conform to the scaling factor of thebackground image 16 d.

Once the texturized image has been completely transformed, it can beinserted 18 into the background image in its proper position. Theprocessed buffer is swapped onto the display screen for viewing and thenext live video frame is obtained 19.

FIG. 2 is a sample screen display showing a live video feed integratedinto a contextual map background. The background map 20 shown here is aportion of a street map. The circles 22 a-c represent a live video feedfrom a camera flying overhead. The area of the circle(s) corresponds tothe field of view of the camera. Each circle represents a frame-by-frameshift in the live video feed as the camera flies over an area. Forinstance, circle 22 a corresponds to frame 1, circle 22 b corresponds toframe 2, and circle 22 c corresponds to frame 3. The circles changeheading as the camera changes heading.

In a similar manner, a street map or other image can be texture mappedover the background map to provide a detailed picture of an area. In atwo-dimensional display, this is a simple process of knowing thecoordinates of the image location, drawing a rectangle over thebackground, and texture mapping the image onto the rectangle. In athree-dimensional display, the image could also be textured onto anelevation grid, which allows it to display the gradations of the terrainbeneath and make use of lighting to highlight elevation changes.Currently, the image that is read in is RGB format, but that can beexpanded to include multiple image formats, such as a bitmap (BMP).

The background image maps can be obtained from any number of sourcesincluding, but not limited to, the United States Geological Survey(USGS) maps. For purposes of the present invention, the background mapsare considered data to be imported and used by the processes of thepresent invention. Similarly, the position data of the camera includingits heading information is data to be processed by the presentinvention. Thus, the coordinate system used is not critical to thepresent invention since the coordinate system merely represents datathat can be translated. The present invention can thus be implemented towork with a variety of background images and a variety of coordinatesystems without departing from the spirit or scope of the presentinvention.

It is to be understood that the present invention illustrated herein isreadily implementable by those of ordinary skill in the art as acomputer program product having a medium with a computer programembodied thereon. The computer program product is capable of beingloaded and executed on the appropriate computer processing device(s) inorder to carry out the method or process steps described. Appropriatecomputer program code in combination with hardware implements many ofthe elements of the present invention. This computer code is oftenstored on storage media. This media can be a diskette, hard disk,CD-ROM, optical storage media, or tape. The media can also be a memorystorage device or collection of memory storage devices such as read-onlymemory (ROM) or random access memory (RAM). Additionally, the computerprogram code can be transferred to the appropriate hardware over sometype of data network.

The present invention has been described, in part, with reference toflowchart or logic flow diagrams. It will be understood that each blockof the flowchart diagrams or logic flow diagrams, and combinations ofblocks in the flowchart diagrams or logic flow diagrams, can beimplemented by computer program instructions.

These computer program instructions may be loaded onto a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructionsthat execute on the computer or other programmable data processingapparatus create means for implementing the functions specified in theflowchart block or blocks or logic flow diagrams.

These computer program instructions may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meanswhich implement the function specified in the flowchart blocks or logicflow diagrams. The computer program instructions may also be loaded ontoa computer or other programmable data processing apparatus to cause aseries of operational steps to be performed on the computer or otherprogrammable apparatus to produce a computer implemented process suchthat the instructions which execute on the computer or otherprogrammable apparatus provide steps for implementing the functionsspecified in the flowchart blocks or logic flow diagrams.

Accordingly, block(s) of flowchart diagrams and/or logic flow diagramssupport combinations of means for performing the specified functions,combinations of steps for performing the specified functions and programinstruction means for performing the specified functions. It will alsobe understood that each block of flowchart diagrams and/or logic flowdiagrams, and combinations of blocks in flowchart diagrams and/or logicflow diagrams can be implemented by special purpose hardware-basedcomputer systems that perform the specified functions or steps, orcombinations of special purpose hardware and computer instructions.

In the following claims, any means-plus-function clauses are intended tocover the structures described herein as performing the recited functionand not only structural equivalents but also equivalent structures.Therefore, it is to be understood that the foregoing is illustrative ofthe present invention and is not to be construed as limited to thespecific embodiments disclosed, and that modifications to the disclosedembodiments, as well as other embodiments, are intended to be includedwithin the scope of the appended claims. The invention is defined by thefollowing claims, with equivalents of the claims to be included therein.

1. A method of integrating a smaller image obtained from a movable videosource moving along a heading into a larger background image that islarger than, and encompasses the field of view of, the video source,said method comprising: texture mapping the smaller image onto asurface; translating the texture mapped image so that it is centeredaround the view point of the moving video source; translating thetexture mapped image to the correct position relative to the largerbackground image by using position data pertaining to the moving videosource; rotating the texture mapped image so that it is tangent to thelarger background image; rotating the texture mapped image so that it isaligned with the heading of the moving video source; scaling the texturemapped image so that it is the same scale as the larger backgroundimage; and inserting, without blending, the texture mapped image intothe larger background image.
 2. A system for integrating a smaller imageobtained from a movable video source moving along a heading into alarger background image that is larger than, and encompasses the fieldof view of, the video source, said system comprising: a processing unitcoupled with said moving video source for; receiving video images andposition data with respect to the source of the video images; texturemapping said video images; and performing matrix transformations on saidtexture mapped video images in order to manipulate the texture mappedvideo images so as to be insertable into a larger background image,wherein said matrix transformations comprise: translating the texturemapped image so that it is centered around the view point of the movingvideo source; translating the texture mapped image to the correctposition relative to the larger background image by using position datapertaining to the moving video source; rotating the texture mapped imageso that it is tangent to the larger background image; rotating thetexture mapped image so that it is aligned with the heading of themoving video source; and scaling the texture mapped image so that it isthe same scale as the larger background image, and a screen displaycoupled with said processing unit for displaying a combined texturemapped video image inserted into the larger background image, withoutblending.
 3. A method of integrating a video feed obtained from amovable aerial video source moving along a heading into a largerbackground map image, where the background map image is larger than, andencompasses the field of view of, the movable aerial video source, saidmethod comprising: texture mapping each frame of the video feed onto asurface; translating the texture mapped frames so that they are centeredaround the view point of the moving aerial video source; translating thetexture mapped frames to the correct position relative to the largerbackground image by using position data pertaining to the moving videosource; rotating the texture mapped frames so that they are tangent tothe larger background map image; rotating the texture mapped frames sothat they are aligned with the heading of the moving aerial videosource; scaling the texture mapped frames so that they are the samescale as the larger background map image; and inserting, withoutblending, the texture mapped frames into the larger background mapimage.
 4. The method of claim 3 wherein said translating the texturemapped frames to the correct position relative to the larger backgroundimage step utilizes precise position data pertaining to the movingaerial video source that is synchronized with each frame of the videofeed.
 5. A system for integrating a video feed obtained from a movableaerial video source moving along a heading into a larger background mapimage, where the background map image is larger than, and encompassesthe field of view of, the movable aerial video source, said systemcomprising: a processing unit wirelessly coupled to said moving videosource for: receiving a video feed; receiving position data with respectto the source of the video feed; texture mapping each frame of the videofeed; and performing matrix transformations on said texture mapped videoframes in order to manipulate the texture mapped video frames so as tobe insertable into the larger background map image, wherein said matrixtransformations comprise: translating the texture mapped frames so thatthey are centered around the view point of the movable aerial videosource; translating the texture mapped frames to the correct positionrelative to the larger background map image by using position datapertaining to the moving video source; rotating the texture mappedframes so that they are tangent to the larger background map image;rotating the texture mapped frames so that they are aligned with theheading of the moving aerial video source; and scaling the texturemapped frames so that they are the same scale as the larger backgroundmap image, and a screen display coupled with said processing unit fordisplaying combined texture mapped video frames inserted into the largerbackground map image, without blending.
 6. The system of claim 5 whereinsaid translating the texture mapped frames to the correct positionrelative to the larger background image utilizes precise position datapertaining to the movable moving aerial video source that issynchronized with each frame of the video feed.
 7. A computer programproduct for integrating a smaller image obtained from a movable videosource moving along a heading into a larger background image that islarger than, and encompasses the field of view of, the video source, thecomputer program product having a medium with a computer programembodied thereon, said computer program product comprising: computerprogram code for texture mapping the smaller image onto a surface;computer program code for translating the texture mapped image so thatit is centered around the view point of the moving video source;computer program code for translating the texture mapped image to thecorrect position relative to the larger background image by usingposition data pertaining to the moving video source; computer programcode for rotating the texture mapped image so that it is tangent to thelarger background image; computer program code for rotating the texturemapped image so that it is aligned with the heading of the moving videosource; computer program code for scaling the texture mapped image sothat it is the same scale as the larger background image; and computerprogram code for inserting, without blending, the texture mapped imageinto the larger background image.